Skip to content

[fix](sc) Skip empty rowset version hole filling for versions <= alter_version#56209

Merged
liaoxin01 merged 1 commit intoapache:masterfrom
liaoxin01:fix_empty_rowset_sc
Sep 19, 2025
Merged

[fix](sc) Skip empty rowset version hole filling for versions <= alter_version#56209
liaoxin01 merged 1 commit intoapache:masterfrom
liaoxin01:fix_empty_rowset_sc

Conversation

@liaoxin01
Copy link
Contributor

@liaoxin01 liaoxin01 commented Sep 18, 2025 •
edited
Loading

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Schema change and version hole handling:

  • Improved the logic in CloudMetaMgr::fill_version_holes to selectively skip filling version holes for schema change tablets (TABLET_NOTREADY state) when the version is less than or equal to alter_version. This prevents abnormal compaction scores and unexpected errors during schema changes, allowing the process to proceed smoothly.
  • In CloudSchemaChangeJob::_process_delete_bitmap, modified the handling of rowsets during schema change jobs to retain only the [0-1] version rowset and set the alter_version for the temporary tablet, ensuring that hole rowsets are only filled for versions greater than alter_version.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 18, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Sep 18, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 18, 2025

TPC-H: Total hot run time: 1476 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d014068b984e7682d2f5c4f8655863ac2727f923, data reload: false

------ Round 1 ----------------------------------
q1	17617	35	19	19
q2	2630	25	24	24
q3	10722	16	16	16
q4	11188	17	15	15
q5	8113	17	14	14
q6	229	12	11	11
q7	957	25	21	21
q8	9461	13	11	11
q9	9624	22	12	12
q10	7010	12	11	11
q11	721	20	18	18
q12	340	13	11	11
q13	18633	12	11	11
q14	231	11	9	9
q15	602	13	10	10
q16	1041	993	958	958
q17	563	13	12	12
q18	7808	13	11	11
q19	1419	12	12	12
q20	1985	388	238	238
q21	9109	21	23	21
q22	1124	13	11	11
Total cold run time: 121127 ms
Total hot run time: 1476 ms

----- Round 2, with runtime_filter_mode=off -----
q1	12	10	10	10
q2	19	19	19	19
q3	11	11	11	11
q4	12	11	11	11
q5	11	11	14	11
q6	11	11	10	10
q7	19	19	19	19
q8	11	9	10	9
q9	10	11	10	10
q10	10	9	10	9
q11	19	19	19	19
q12	11	11	10	10
q13	10	10	10	10
q14	10	11	13	11
q15	11	10	10	10
q16	1047	1097	1038	1038
q17	12	12	10	10
q18	10	9	9	9
q19	10	10	10	10
q20	1862	1977	1839	1839
q21	20	19	20	19
q22	11	10	10	10
Total cold run time: 3159 ms
Total hot run time: 3114 ms

@doris-robot
Copy link

doris-robot commented Sep 18, 2025

TPC-DS: Total hot run time: 2785 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d014068b984e7682d2f5c4f8655863ac2727f923, data reload: false

query1	1071	20	10	10
query2	7160	20	16	16
query3	7635	12	11	11
query4	26560	12	11	11
query5	4488	12	12	12
query6	374	12	11	11
query7	5431	12	12	12
query8	362	20	17	17
query9	9217	12	11	11
query10	730	12	11	11
query11	15648	11	11	11
query12	176	11	10	10
query13	1696	12	11	11
query14	10947	15	13	13
query15	222	13	11	11
query16	7187	12	11	11
query17	1531	11	11	11
query18	2803	12	11	11
query19	213	10	9	9
query20	139	11	9	9
query21	223	12	12	12
query22	4033	10	9	9
query23	33567	18	12	12
query24	9578	13	13	13
query25	719	10	9	9
query26	1100	10	9	9
query27	3376	10	9	9
query28	6370	11	9	9
query29	1216	11	9	9
query30	643	10	9	9
query31	1758	11	10	10
query32	120	11	9	9
query33	1311	11	9	9
query34	1689	811	515	515
query35	875	11	10	10
query36	999	11	9	9
query37	121	10	9	9
query38	3527	9	8	8
query39	1550	733	770	733
query40	239	10	13	10
query41	81	11	10	10
query42	153	10	8	8
query43	509	11	10	10
query44	1371	10	9	9
query45	194	10	10	10
query46	905	10	8	8
query47	1827	9	9	9
query48	402	10	10	10
query49	1155	11	9	9
query50	758	10	8	8
query51	3959	10	9	9
query52	111	10	9	9
query53	234	10	11	10
query54	747	10	9	9
query55	96	9	9	9
query56	333	11	8	8
query57	1170	8	8	8
query58	306	9	8	8
query59	2629	9	9	9
query60	353	10	9	9
query61	179	9	8	8
query62	844	9	8	8
query63	269	11	11	11
query64	4240	10	9	9
query65	4014	9	8	8
query66	1084	11	11	11
query67	16419	11	10	10
query68	3451	9	8	8
query69	646	10	9	9
query70	1327	10	10	10
query71	421	350	360	350
query72	6803	12	9	9
query73	546	10	11	10
query74	9684	13	13	13
query75	3507	10	9	9
query76	2705	12	10	10
query77	846	13	12	12
query78	10018	21	10	10
query79	1154	12	12	12
query80	763	15	13	13
query81	534	11	11	11
query82	317	13	13	13
query83	411	11	11	11
query84	348	10	9	9
query85	1628	11	11	11
query86	509	11	10	10
query87	3884	10	9	9
query88	2895	15	11	11
query89	411	10	11	10
query90	2235	10	10	10
query91	173	10	8	8
query92	82	10	10	10
query93	1060	10	10	10
query94	766	9	9	9
query95	518	10	8	8
query96	435	10	10	10
query97	3094	8	9	8
query98	304	244	236	236
query99	1505	11	11	11
Total cold run time: 283926 ms
Total hot run time: 2785 ms

@doris-robot
Copy link

doris-robot commented Sep 18, 2025

ClickBench: Total hot run time: 0.06 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d014068b984e7682d2f5c4f8655863ac2727f923, data reload: false

query1	0.06	0.01	0.00
query2	0.10	0.00	0.01
query3	0.26	0.00	0.01
query4	1.75	0.00	0.01
query5	0.28	0.01	0.00
query6	1.64	0.01	0.00
query7	0.05	0.01	0.00
query8	0.06	0.01	0.00
query9	0.62	0.01	0.00
query10	0.60	0.01	0.01
query11	0.17	0.00	0.00
query12	0.16	0.00	0.01
query13	0.65	0.00	0.00
query14	1.05	0.00	0.00
query15	0.90	0.00	0.01
query16	0.40	0.01	0.00
query17	1.09	0.01	0.01
query18	0.22	0.02	0.01
query19	1.97	0.00	0.00
query20	0.02	0.01	0.00
query21	15.36	0.01	0.00
query22	6.85	0.00	0.01
query23	15.77	0.00	0.00
query24	1.39	0.01	0.01
query25	0.22	0.00	0.00
query26	0.17	0.00	0.00
query27	0.11	0.00	0.00
query28	1.11	0.00	0.00
query29	13.26	0.00	0.00
query30	0.32	0.00	0.00
query31	2.26	0.01	0.00
query32	5.86	0.01	0.00
query33	4.36	0.00	0.00
query34	7.31	0.00	0.00
query35	5.97	0.00	0.00
query36	0.69	0.00	0.00
query37	0.12	0.00	0.00
query38	0.07	0.01	0.00
query39	0.04	0.00	0.00
query40	0.19	0.01	0.01
query41	0.09	0.00	0.00
query42	0.06	0.01	0.01
query43	0.05	0.00	0.01
Total cold run time: 93.68 s
Total hot run time: 0.06 s

@doris-robot
Copy link

doris-robot commented Sep 18, 2025

BE UT Coverage Report

Increment line coverage 38.24% (13/34) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.34% (17593/33616)
Line Coverage 37.54% (159705/425404)
Region Coverage 32.06% (121559/379170)
Branch Coverage 33.43% (53360/159617)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 18, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (34/34) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.04% (23340/32855)
Line Coverage 57.36% (243519/424511)
Region Coverage 52.84% (203104/384346)
Branch Coverage 54.42% (87267/160354)

dataroaring
dataroaring approved these changes Sep 19, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 19, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 19, 2025

PR approved by anyone and no changes requested.

@liaoxin01 liaoxin01 merged commit 944b4e5 into apache:master Sep 19, 2025
29 of 32 checks passed
@liaoxin01 liaoxin01 deleted the fix_empty_rowset_sc branch September 19, 2025 02:58
liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Sep 19, 2025
@liaoxin01
[fix](sc) Skip empty rowset version hole filling for versions <= alte…
6b0692e 
…r_version (apache#56209)

**Schema change and version hole handling:**

* Improved the logic in `CloudMetaMgr::fill_version_holes` to
selectively skip filling version holes for schema change tablets
(`TABLET_NOTREADY` state) when the version is less than or equal to
`alter_version`. This prevents abnormal compaction scores and unexpected
errors during schema changes, allowing the process to proceed smoothly.
* In `CloudSchemaChangeJob::_process_delete_bitmap`, modified the
handling of rowsets during schema change jobs to retain only the [0-1]
version rowset and set the `alter_version` for the temporary tablet,
ensuring that hole rowsets are only filled for versions greater than
`alter_version`.
@liaoxin01 liaoxin01 mentioned this pull request Sep 19, 2025
Merged
morrySnow pushed a commit that referenced this pull request Sep 19, 2025
@liaoxin01
branch-3.1: [fix](sc) Skip empty rowset version hole filling for vers…
fefeaf0 
…ions <= alter_version #56209 (#56212)

cherry pick from #56209
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@luwei16 luwei16 luwei16 approved these changes

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@liaoxin01 @hello-stephen @doris-robot @luwei16 @dataroaring @morrySnow